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Abstract 

Background Improving children’s oral language skills is an important focus of educa¬ 
tional research and practice; however, relatively few interventions have demonstrated 
impacts on these skills. This work makes a unique contribution to our understanding of the 
effects of language-focused interventions in pre-kindergarten settings by examining 
impacts on both lower- and higher-level language skills as well as overall language 
comprehension. 

Objective The goal is to assess the impacts of business-as-usual pre-kindergarten with 
implementation of two versions of an experimental curriculum supplement, Let’s Know!, 
designed to enhance three component language skills (vocabulary, comprehension moni¬ 
toring, and text-structure knowledge) and overall language comprehension in pre¬ 
kindergarteners. 

Methods Eleven pre-kindergarten teachers and 49 low socioeconomic-status students 
participated. Teachers were randomly assigned to either business-as-usual, Let’s Know! 
Broad, or Let’s Know! Deep, unless they participated in a previous pilot study, in which 
case they were randomly assigned to either Let’s Know! Broad or Deep. The Broad version 
included five different lesson types, whereas the Deep version included three lesson types 
with additional practice. Children’s gains were assessed proximally with measures of 
vocabulary, comprehension monitoring, and text-structure knowledge and distally with a 
measure of language comprehension. 

Results Children in both experimental versions significantly improved their vocabulary 
skills relative to children who received business-as-usual instruction. For comprehension 
monitoring, children who received the Deep and Broad versions improved their scores 
relative to BAU children for Units 1 and 3, respectively. Improvement in language 
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comprehension was only found for children who received Let's Know! Deep compared 
with business-as-usual. 

Conclusion This study provides initial evidence that the Let's Know! curricula may 
serve to foster young children’s vocabulary, comprehension monitoring, and language 
comprehension skills. 

Keywords Oral language intervention • Vocabulary • Comprehension monitoring • Text- 
structure • Pre-kindergarten 


Introduction 

During the last decade, a large number of classroom-based language-focused programs and 
practices have been developed and tested for their impacts on pre-kindergarten (pre-K) 
children’s oral language skills (Assel et al. 2007; Jackson et al. 2006; Justice et al. 2008; 
Lonigan et al. 2013; Mashburn et al. 2010; Piasta et al. 2010; Powell et al. 2010). Some of 
these interventions have involved provision of intensive professional development (PD) for 
pre-K teachers to promote their use of specific language-facilitating practices (e.g., Piasta 
et al. 2010; Powell et al. 2010), whereas others featured teachers’ implementation of 
curricula providing an explicit scope and sequence of language-focused instruction (e.g., 
Justice et al. 2008; Lonigan et al. 2013). Regardless of approach, interests in improving 
young children’s oral language skills stem from the well-established relations between 
children’s early language skills and future academic achievement, particularly in reading 
comprehension (Storch and Whitehurst 2002). If children’s language skills can be 
enhanced during the pre-K year, it stands to reason that this might have positive effects on 
children’s future academic performance as a result. 

An important finding from the language-intervention literature to date is that relatively 
few interventions appear successful in significantly elevating the language skills of young 
children. For instance, Powell and his colleagues studied the language growth of children 
in the classrooms of pre-K teachers who participated in a semester-long PD course 
involving workshop training plus in-class coaching. A major emphasis of the PD was to 
improve teachers’ use of specific language-facilitating practices in the classroom, such as 
providing expansions of children’s language use during conversations and read-alouds 
(Powell et al. 2010). A comparison of language growth over the academic year of children 
whose teachers who did not participate in the PD showed no advantage for those children 
whose teachers received extensive training in language-facilitating practices (d = 0.03). 
Other studies have reported similar findings, with teachers’ use of specific curricula and/or 
receipt of PD having limited effects on children’s language skills (Bierman et al. 2008; 
Cabell et al. 201 1). Bierman et al. (2008) studied the effects of a multi-component enriched 
curriculum, including language-enrichment activities, for children in Head Start; the 
intervention had little discernable effects on children’s language skills (ds = —0.07 to 
0.15). Likewise, Cabell et al. (2011) studied the effects of teachers’ use of language- 
facilitating strategies they learned in PD for at-risk preschoolers; effects of the intervention 
on children’s language skills was also quite modest (ds = 0.05-0.13). 

Meta-analyses of vocabulary-specific interventions have found few specific intervention 
features that appear to contribute to large impacts on child outcomes, components of which 
included teacher or experimenter implementation, or use of explicit or implicit instruction 
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(Marulis and Neuman 2010, 2013). Indeed, few interventions were able to reduce a pov¬ 
erty-induced achievement gap. However, some vocabulary interventions focused on low- 
SES populations have found a measure of success. For example, a vocabulary intervention 
with kindergarten and first grade students from a low-SES school found that when 
sophisticated words were introduced during book reading and discussed afterward, children 
learned the vocabulary words more than those words not discussed after (Beck and 
McKeown 2007). Moreover, children learned better with six rather than three days of 
working with the sophisticated words. Another study with kindergarten students from a 
low-SES area found that teaching vocabulary words embedded in a story resulted in longer 
lasting vocabulary knowledge than incidental vocabulary learning (Coyne et al. 2009). 
However, teaching vocabulary using an “extended” strategy of spreading the teaching 
across several days resulting in a deeper understanding of the vocabulary words compared 
to an embedded strategy. In general, other factors relating to vocabulary interventions that 
have been found to have larger effects on children include having a teacher or researcher 
implement the intervention and using proximal, author created interventions rather than 
distal, standardized interventions (Marulis and Neuman 2013). Additionally, low SES was 
associated with smaller effect sizes for vocabulary interventions. 

The current study adds to the literature reviewed by Marulis and Neuman (2010, 
2013) by investigating a teacher-implemented language-focused intervention on cur¬ 
riculum-based assessments closely aligned to the intervention, as well as a standardized 
language comprehension measure, and which featured both explicit and implicit 
instruction. We targeted several of the intervention components previously discussed 
which have generally been found to relate to low (i.e., working with low-SES students, 
using a distal measure of vocabulary) and high effect sizes (i.e., also using a proximal 
measure of vocabulary), in an effort to further explore the discrepancy between effect 
sizes for the different types of measures and components of vocabulary interventions. 
Moreover, we also delve deeper into the implementation schedule than just frequency or 
intensity of teaching by testing two variations of the intervention. As noted above, Coyne 
et al. (2009) found different advantages of using a broad/embedded teaching style 
compared to a deep/extended strategy. Here, we piece apart these components, use both 
proximal and distal measures, and focus on a low-SES population greatly in need of 
vocabulary support. 

An important distinction of the intervention examined here and those evaluated by 
Marulis and Neuman is that the present experimental intervention focused not only on 
vocabulary but also gains in children’s text-structure knowledge, comprehension moni¬ 
toring, and general language comprehension skills. Although some studies have found 
successful explicit teaching strategies for improving text-structure knowledge and com¬ 
prehension, these studies have primarily focused on 7- to 8-year-old children (Williams 
et al. 2005; Williams et al. 2009). We intend to build upon these findings by working 
with younger children. Specifically, in the present study we report results of a year-long 
pilot study designed to examine impacts on children’s oral language skills when their 
pre-K teachers implemented an experimental curriculum supplement, Let’s Know!. Let’s 
Know! is intended to be embedded within the typical language-arts/reading curriculum 
and provide enhanced opportunities for children to develop their language skills. 
Development of the supplement, which was the major task of a federally funded research 
consortium, is detailed extensively under separate cover (Language and Research 
Research Consortium, in press), thus we only highlight features of its development here. 
Let’s Know! was developed and revised over a two-year iterative research process in 
which the major goal was to ensure that teachers could use the supplement with fidelity. 
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Each component of the intervention, such as the organization of lessons and the use of 
specific materials, was studied over the course of several design studies employing 
various methodologies (e.g., systematic observations, interviews, and focus groups) to 
ensure that it could be implemented as intended. When teachers’ fidelity to any com¬ 
ponent was low or poor, it was revised and retested. Consequently, the final draft of the 
supplement was one that teachers could implement with fidelity, including following the 
lessons as intended, enacting the scope and sequence, and using specialized materials 
designed for the intervention (e.g., graphic organizers). The research presented here is 
the first test of the potential impacts of teachers’ implementation of Let’s Know! on 
children’s language skills, specifically on those language skills that contribute to 
enhanced listening comprehension and, theoretically, reading comprehension. 

To this point, Let’s Know! was designed to provide pre-K teachers an instructional 
tool to explicitly improve children’s lower- and higher-level language skills and overall 
language comprehension. Although not a focus of this study, there are also versions of 
Let’s Know! for kindergarten through third grade students as well as a pre-K bilingual 
version. The focus of the supplement on improving children’s language skills is 
grounded in the Simple View of Reading, which proposes reading comprehension to be 
the multiplicative function of decoding and language comprehension (Gough and Tunmer 
1986). Both theory and research demonstrate that among skilled readers, one’s skills in 
reading comprehension approximate one’s language comprehension (Perfetti 2007). 
Among the most critical determinants of language comprehension and in turn, reading 
comprehension, are an individual’s language skills, transcending component skills that 
can be characterized as lower- (i.e., automatic) and higher-level (i.e., integrative) (Catts 
et al. 2006). 

Lower-level language skills influential to language comprehension include grammar 
(i.e., syntax and morphology) and vocabulary (Verhoeven and Van Leeuwe 2008), which 
together represent automatic processes that directly support one’s ability to comprehend 
when they hear (i.e., language comprehension) and read (i.e., reading comprehension) 
(Perfetti 2007, p. 358). Higher-level language skills, on the other hand, represent inte¬ 
grative processes that allow one to infer, to monitor and correct one’s comprehension, 
and to organize information coherently. Higher-level language skills particularly influ¬ 
ential to skilled comprehension include inferencing, comprehension monitoring, and text- 
structure knowledge. Inferencing refers to one’s ability to fill in missing information to 
arrive at a coherent, integrated representation of discourse heard or read; for instance, 
children often must infer the goals or motives of characters in stories (Lynch and van 
den Broek 2007). Comprehension monitoring refers to the ability to reflect on one’s own 
comprehension and, in instances when comprehension is compromised (e.g., information 
read or heard is inconsistent), to seek to repair it (Wagoner 1983). Text-structure 
knowledge refers to one’s knowledge about how texts are organized and includes one’s 
ability to use critical information contained in texts to ascertain the type of text being 
read or heard, such as key words that signal cause-effect, compare-contrast, and 
sequences (Cain et al. 2004). These higher-level language skills are collectively referred 
to as “higher-level meaning construction skills” and “higher-level factors in compre¬ 
hension” (respectively, Cain et al. 2004; Perfetti 2007) and, along with lower-level 
language skills (e.g., vocabulary, grammar), represent the scope of language-focused 
instruction addressed in Let’s Know!, based on the hypothesis that promoting these 
component skills will lead to improvements in overall language comprehension in the 
short-term and reading comprehension in the long-term, although in this study we only 
report language skills upon completion of the intervention. In the present work, we 
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examine impacts of Let’s Know! implementation by pre-K teachers on a subset of these 
lower- and higher-level language skills (viz., vocabulary, comprehension monitoring, 
text-structure knowledge), representing desired proximal outcomes, as well as general 
language comprehension, representing the desired distal outcome. 

Although a number of recent studies have examined the effects of language-focused 
interventions within pre-K settings, these have largely if not entirely focused on 
improving children’s lower-level language skills, such as grammar and vocabulary (Assel 
et al. 2007; Jackson et al. 2006; Justice et al. 2008; Lonigan et al. 2013; Mashburn et al. 
2010; Piasta et al. 2010; Powell et al. 2010). However, the importance of higher-level 
language skills to overall language comprehension and future reading comprehension 
suggests that these too should receive explicit attention in language-focused interven¬ 
tions. In fact, it may be that children’s higher-level language skills serve as a critically 
important causal lever for enhancing children’s overall language comprehension, as 
suggested in several recent studies involving children in the primary grades (Clarke et al. 
2010; Williams 2005; Williams et al. 2004, 2005). Clarke and colleagues, for instance, 
tested the effects of a language-focused intervention that featured activities designed to 
boost both lower- (e.g., vocabulary) and higher-level skills (e.g., text-structure knowl¬ 
edge, inferencing) for 8- and 9-year-old children with poor language and reading com¬ 
prehension. Compared to more traditional forms of reading instruction (e.g., strategy 
training) and no instruction (the business-as-usual condition), those who received the 
language-focused intervention had improved language skills and reading comprehension 
1 year post-intervention. 

The purpose of this study was to examine the impacts of Let’s Know! on the lan¬ 
guage skills of pre-K children. This work makes an incremental and unique contribution 
to our understanding of the effects of language-focused interventions in pre-K settings in 
that it examines impacts on both lower- and higher-level language skills as well as 
overall language comprehension. Although some work has shown that children’s lan¬ 
guage skills, such as vocabulary and grammar, can be improved through targeted 
interventions delivered in pre-K settings (e.g., Penno et al. 2002; Pollard-Durodola et al. 
2011; Wasik and Bond 2001), studies have seldom included general measures of lan¬ 
guage comprehension as outcome measures, nor have they included measures of chil¬ 
dren’s higher-level language skills, such as comprehension monitoring and text-structure 
knowledge. This work helps to determine whether such skills can be improved through 
classroom-based interventions. To this end, this study addressed two research questions. 
First, to what extent does pre-K teachers’ implementation of Let’s Know! have positive 
impacts on children’s lower- and higher-level language skills, specifically vocabulary, 
comprehension monitoring, and text-structure knowledge, compared to typical classroom 
practices? We hypothesized that children who were exposed to Let’s Know! would 
significantly outperform children receiving typical classroom practices on measures of 
these skills post-intervention. Second, to what extent does pre-K teachers’ implemen¬ 
tation of Let’s Know! impact children’s general language comprehension, as compared 
to typical classroom practices? We hypothesized that children exposed to Let’s Know! 
would significantly outperform children receiving typical classroom practices on a 
general measure of language comprehension post-intervention. Should these hypotheses 
be confirmed, the results of this study can benefit the field substantially by identifying 
an efficacious avenue for improving young children’s lower- and higher-level language 
skills as well as general language comprehension within the preschool classroom 
context. 
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Methods 

Participants 

Eleven pre-K teachers in two states and a subset of children in their classrooms served as 
participants in this study. Participants were recruited prior to the 2012-2013 school year 
and received incentives in the form of checks or gift cards to teachers for participating. All 
of the teachers were lead teachers in publicly funded pre-K classrooms, all of which 
targeted enrollment towards children from low-socioeconomic status (SES) backgrounds. 
Teachers self-selected to enroll in the study following participation in information sessions 
providing details of the project. Twelve teachers initially enrolled in the study, but one 
subsequently dropped out after random assignment. Table 1 provides details regarding the 
gender, race, educational background, and teaching experience for the remaining teachers. 
As can be seen, the 11 teachers were all female, were generally well-educated (i.e., all had 
at least a Bachelor’s degree), and were largely Caucasian. Informed consent was collected 
for all participants and Institutional Review Board approval was obtained. Authors report 
no conflict of interest. 

In each classroom, up to five children were randomly selected to participate in ongoing 
assessments designed to measure intervention impacts. To identify this subset of children, 
caregivers of all children in each classroom were given a study brochure and informed- 
consent agreement in the fall of the academic year. Eligibility criteria included that the 
child must be proficient in English, have no severe or profound disabilities, attend class at 
least 3 days a week, and be older than 3 years and 9 months of age. From the consents 
received and deemed eligible, up to five children were randomly selected from each 
classroom or, in cases in which only five consents were received, all children were 


Table 1 Descriptive informa¬ 
tion for participating teachers 
(n = 11) 


Years of experience refers to 
years teaching at the pre¬ 
kindergarten/preschool level 


Variable 

Let’s Know! 






BAU 


Broad 


Deep 



N 

% 

N 

% 

N 

% 

Education level 

BA 

3 

75 

1 

33 

1 

25 

1+ year beyond BA 

0 

0 

0 

0 

2 

50 

Master’s 

1 

25 

2 

67 

1 

25 

Gender 

Female 

4 

100 

3 

100 

4 

100 

Race 

African American 

0 

0 

0 

0 

0 

0 

American Indian 

1 

25 

0 

0 

0 

0 

White/Caucasian 

3 

75 

3 

100 

4 

100 

Years of experience 

1 

2 

50 

0 

0 

0 

0 

2 

0 

0 

0 

0 

0 

0 

3-4 

0 

0 

0 

0 

1 

25 

5-10 

0 

0 

0 

0 

2 

50 

11 + 

2 

50 

3 

100 

1 

25 


Springer 










Child Youth Care Forum (2016) 45:367-392 


373 


selected. In total, 49 children were included (25 males, 16 females, 8 unreported). Their 
mean age in the fall of the year was 53 months ( SD = 3.79, range 45.21-59.6). In terms of 
race, 57 % of the children were White/Caucasian, 12 % were Black/African-American, 
14 % were Asian, and 17 % were other ethnicities or the information was unreported. Data 
about disabilities were missing for nine children. Of the remaining 40 children, 12.5 % had 
identified disabilities and 8 % had difficulties in sensory or cognitive functioning, based on 
parental report. All of the children were from low-SES backgrounds, based on the targeted 
enrollment practices of participating centers. Accordingly, the parent-reported highest 
level of maternal education for the participating children was relatively low, with only 
29 % of children’s mothers having a bachelor’s degree or higher. Data were missing for 11 
parents. 

Assignment to Study Conditions 

In this study, teachers were assigned to implement one of two versions of Let’s Know! (see 
“Methods and Materials”), which differed in the breadth of language skills targeted within 
the supplement or a control business-as-usual (BAU) condition. In the Broad Version, all 
five component language skills referenced previously were targeted (i.e., grammar, 
vocabulary, inferencing, comprehension monitoring, and text-structure knowledge), 
whereas in the Deep version only three component language skills were targeted (i.e., 
vocabulary, inferencing, and comprehension monitoring). The teacher who left the study 
was in the Broad condition, thus the final number of teachers was as follows: Broad, n = 3; 
Deep, n = 4; and BAU, n = 4. Condition assignment was not wholly random, as several 
teachers had previously participated in the design research developing early drafts of the 
curriculum. These teachers were automatically assigned to implement the experimental 
curriculum, although assignment to the Broad versus Deep versions was random. For the 
remaining teachers, random assignment to one of the three conditions was used. Children’s 
condition assignment was based on their teachers’ assigned condition. Final numbers of 
children per condition were as follows: Broad, n = 15; Deep, n = 18; and BAU, n = 16. 

Methods and Materials 

The primary methods of this study were twofold: (a) implementation of assigned study 
conditions by pre-K teachers, and (b) administration of assessments to children. 

Implementation of Assigned Study Conditions 

The pre-K teachers assigned to the two experimental conditions implemented three units of 
Let’s Know! for 21 weeks of the academic year. Note that the final draft of the curricular 
supplement involves four units; however, this pilot study involved implementation of only 
three units as the fourth unit was not yet complete. Over the same period, BAU teachers 
implemented their typical classroom practices, while completing study-related activities 
designed to reduce Hawthorne effects (e.g., receipt of study incentives, provision of 
classroom materials, and provision of “neutral” professional development). 
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Let’s Know! Implementation 

Irrespective of version, teachers’ implementation of Let’s Know! features 120 min of 
systematic and explicit language-focused instruction each week. Instruction is organized 
into four 30-min whole-class lessons, which are arranged into three seven-week units. The 
units are thematically organized to address a specific topic (Unit 1, Animals; Unit 2, 
Fiction; Unit 3, Earth Materials), and each focuses on a specific type of text-structure (Unit 
1, Compare-Contrast; Unit 2, Sequences and Cycles; Unit 3, Description). Teachers 
received a manual for each unit in its entirety and any materials needed for implementation 
(e.g., trade storybooks, manipulatives), which are described below. 

To implement a Let’s Know! unit, teachers follow semi-scripted lessons which provide 
specific language-focused objectives targeted within a given lesson, a structured sequence 
of activities to be followed for targeting these objectives, and sample scripts for teachers to 
follow. See Appendix 1 for a sample Words to Know lesson plan. Each unit comprises 
different types of lessons that target specific language skills: Text Mapping lessons target 
skills specific to text-structure knowledge and grammar, Words to Know lessons target 
skills specific to vocabulary development, and Integration lessons target skills specific to 
inferencing and comprehension monitoring. The Words to Know lessons involve providing 
specific definitions of new words as well as activities designed to build a deeper under¬ 
standing of the new words. Text Mapping and Integration lessons focus on developing 
more general understandings of language, taught less explicitly than the vocabulary words 
in the Words to Know lessons. Two additional lesson types provide children with 
opportunities to participate in whole-class read-alouds with their teacher (i.e., Read to Me) 
and to explore books on their own (i.e., Read to Know). 

The two versions of Let’s Know! differ in relation to the types of lessons implemented. 
As shown in Table 2, the Broad version features all five lesson types organized into a 
seven-week sequence of instruction (see Appendix 2 for full schedule). The Deep version 
is also seven-weeks, however, the distinguishing feature of the Deep version is that there 
are fewer lesson types occurring in each unit but also increased opportunities for children 
to practice certain skills. The development of the Deep version (which is derived from the 
Broad version) was a result of teacher input during the curriculum design studies. 
Specifically, it addresses some teachers’ suggestions that the intervention would be more 
effective if students had opportunities to practice skills specific to vocabulary, inferencing, 
and text-structure knowledge. It features three lesson types, omitting the Text Mapping and 
Read to Know lessons. In the Deep version, the Words to Know and the Integration lessons 


Table 2 Lesson types in Let’s Know! 


Lesson type 

Skills targeted 

Version 

Broad 

Deep 

Words to know 

Vocabulary 

X 

X 

Integration 

Inferencing, comprehension monitoring 

X 

X 

Text mapping 

Text-structure knowledge, grammar 

X 


Read to know 

Motivation, background knowledge 

X 


Read to me 

Motivation, background knowledge 

X 

X 
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are expanded upon to provide increased opportunities to develop vocabulary (i.e., Words to 
Know), inferencing, and comprehension monitoring (i.e., Integration) skills. These 
expanded lessons (Words to Know Practice, Integration Practice) are similar to the original 
Words to Know or Integration lessons. 

To support teachers’ implementation of both the Broad and Deep versions of Let’s 
Know!, a number of strategies were used. First, researchers met individually with each 
participating teacher prior to the start of the year to go over the manual of lesson plans, 
describe each of the lesson types, and to orient them to professional development (PD) 
modules that they would watch online. The manual contained all lessons and their 
implementation schedules, materials for each lesson, an overview of teaching strategies 
specific to each lesson type, and other necessary study materials (e.g., consent forms, 
contact information, list of participants). In addition, the PD modules (of which there were 
3-8 depending on condition) provided a more in-depth examination of each of the lesson 
types to be used; these included videos of teachers implementing the lesson types and a 
moderator describing characteristics of effective lesson implementations. Let’s Know! 
teachers completed the modules, which took approximately 6 h, prior to implementing the 
first lesson. After viewing each module, teachers completed an online survey indicating 
their comfort level of the given topic covered in the module. The surveys were used to 
monitor completion of the PD by each teacher. As a follow-up to the online training, 
research staff met individually with each teacher 2 weeks into implementation of the first 
unit. In this 30-60 min session, the staff member used information from teachers’ PD 
surveys to identify any possible area in which they might need assistance or additional 
training (e.g., implementing a certain type of lesson). 

Implementation Fidelity 

Implementation fidelity for the Let’s Know! teachers was monitored using classroom 
observations of an entire Let’s Know! lesson, conducted within each classroom twice per 
unit. It is important to note that the development of Let’s Know!, which occurred over a 
two-year period of iterative design studies, emphasized development of an intervention that 
could be used by teachers with fidelity. Thus, examination of fidelity in this pilot study 
largely focused on confirming high levels of fidelity, rather than seeking to explore dif¬ 
ferences in fidelity among teachers. In the present study, select lessons were coded for 
fidelity using a Fidelity Observation Checklist (FOC), developed for this study, which 
recorded teachers’ implementation of the various components of each lesson. Based on live 
observations of 68 lessons in the Broad and 64 lessons in the Deep version, fidelity 
averaged 85 % across lessons. 

Measures 

The measures of relevance to this study are twofold: (a) proximal measures of lower- and 
higher-level language skills, and (b) distal measure of language comprehension. 

Proximal Measures of Language Skills 

A curriculum-based measure (CBM) was given by teachers blind to the other conditions 
and audio-recorded during week six of each seven-week unit in both versions to examine 
children’s skills specific to vocabulary, comprehension monitoring, and text-structure 
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knowledge. Children in the BAU condition also completed these assessments to provide a 
counterfactual. We refer to this as a proximal measure of intervention impacts, as the items 
of the CBM are closely aligned to what is taught within the Let's Know! lessons. The 
vocabulary items required children to provide definitions for eight words targeted during 
the unit’s lessons. An example question is: “Tell me what different means.” Each defi¬ 
nition was scored as correct (2 points), partially correct (1 point), or incorrect (0 points); 
thus, 16 points were possible. 

The comprehension monitoring items required children to listen to a passage, identify 
an inconsistency in the passage, and then identify a strategy to correct the inconsistency. 
For example, the examiner would start reading from the book “Homes of Living Things” 
and purposely replace the word “deer” with the word “tiger” in a passage. After this 
replacement, the examiner would pause to allow the child to spontaneously identify the 
inconsistent item; if the child did not, he was prompted to do so. The child received 2 
points for spontaneous identification of the inconsistency, 1 point for prompted identifi¬ 
cation, and 0 points if the inconsistency was not identified. Then, the examiner asked 
“What kind of fix-up strategy could I use to make sure I’m reading the correct words?” 
The child received one point for identification of a “fix-up” strategy (e.g., re-read the 
passage) and 0 points for an incorrect response. Two passages were administered, and three 
points were possible for each, resulting in a possible total of 6 points. 

The text-structure knowledge items, of which there were two, involved presenting 
children with two short passages. For example, one passage pertained to mice and deer, 
describing ways in which they are alike and different. After listening to each passage, 
children identified the main idea from three options (scored as 1 = correct, 0 = incorrect). 
In this case the main idea was that mice and deer are alike and different. Next, children 
chose the most appropriate title from three options (scored as 1 = correct, 0 = incorrect). 
The appropriate title for the example passage would be Mice and Deer. Other options 
included Animal Food and Animal Size. In total, four points were possible. Note that each 
unit’s CBM was uniquely constructed to examine children’s skills specific to the content of 
a given unit; for instance, vocabulary items on the CBM following Unit 1 (Animals) 
examined children’s knowledge of words targeted in that unit (e.g., shelter, habitat, dif¬ 
ferent, alike). Although the tasks used to assess the three language skills were similar, the 
content varied. For the present purposes, we analyzed children’s scores on the CBMs 
implemented after Unit 1 (week 6 of the intervention) and Unit 3 (week 20 of the inter¬ 
vention), with the latter implemented about 2-3 weeks prior to post-testing on the general 
measure of language comprehension. The Unit 2 CBM is not analyzed here due to highly 
different content material compared to Units 1 and 3, making comparisons between the 
three units ineffective. Specifically, Unit 2 covered fictional stories whereas Units 1 and 3 
covered Animals and Earth Materials, both scientific in nature. 

Distal Measure of Language Comprehension 

The language comprehension measure used at pre- and post-test, referred to as the Lis¬ 
tening Comprehension Measure (LCM), was developed as part of the larger consortium’s 
research activities and administered by trained researchers. The LCM, adapted from the 
Quantitative Reading Inventory-5 (Leslie and Caldwell 2006), provides an evaluation of 
children’s language (rather than reading) comprehension. The LCM was collected at pre- 
and post-test in participating classrooms, in the fall of the year prior to the implementation 
of Let’s Know!, and in the spring of the year, following implementation of the third unit. In 
general, about 22 weeks of instruction spanned pre- to post-test. The measure used is 
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considered distal to the intervention, as it represents a general measure of language 
comprehension and does not examine the specific component language skills targeted in 
the intervention (e.g., vocabulary skills, text-structure knowledge). Children listened to 
each of two passages told orally by an examiner (one narrative, one expository), then they 
were asked to answer six comprehension questions for the first story and four for the 
second. Students also completed a retelling of each story, but the retellings were not 
analyzed for this study. The overall raw score from the comprehension questions is used to 
represent overall performance. Scores on the LCM can range from 0 to 10. The validity of 
the LCM was examined in a consortium-administered longitudinal study involving 416 
pre-K students, who were given the LCM and the Test of Narrative Language (TNL; 
Gillam and Pearson 2004); the correlation between the LCM and TNL was high (0.762). 

Data Analysis 

Analyses to test the impact of Let’s Know! required consideration of the level of measure 
for each outcome, limitations due to sample size, and adjustments for the nested nature of 
the data. This study was based on feasibility to conduct a pilot study, and thus did not 
conduct a power analysis. Language comprehension (measured by LCM), the distal 
measure of language skill, was approximately normally distributed, but responses to the 
three measures of proximal language skill (vocabulary, comprehension monitoring, and 
text structure) on the CBMs did not form continuous scales. Vocabulary was assessed as a 
count of the number of correct responses on definitions of eight words, allowing for partial 
corrections; data from the comprehension monitoring assessment was ordinal and repre¬ 
sented the degree of comprehension and recognition of inconsistencies, for which a larger 
value indicated better comprehension monitoring; and responses regarding text-structure 
were measured as counts of the number of pieces of text information correctly identified 
out of four, total. Consequently, we used a combination of generalized linear modeling 
approaches, including omnibus and contrast tests, confidence intervals, and effect size 
estimates to assess intervention group differences. 

Due to concerns regarding small sample sizes for the non-normal outcomes (Newson 
2001; UCLA: Statistical Consulting Group 2014; Zhu and Lakkis 2014), coupled with only 
partial randomization of teachers to intervention groups, we adjusted for potential residual 
correlation among outcomes for children with the same teacher using generalized esti¬ 
mating equations (GEE) and robust standard errors. GEE analyses are one of the most 
commonly used approaches for longitudinal and clustered data to adjust the standard errors 
in the model for residual correlation (Hilbe 2011). GEE coefficients are interpreted as 
marginal effects; that is, they are averaged over the distribution of random effects for 
teachers and they can be used to reproduce group means (or counts), as we show below. 
When inferences about treatment group differences are desired, rather than inferences 
about individual teacher-specific effects, these marginal or population-averaged coeffi¬ 
cients are particularly useful (Moerbeek et al. 2003). In addition, the clustering of children 
within teachers is appropriately accounted for in GEE analyses, even if the correlational 
structure is mis-specified (Moerbeek et al. 2003). Thus, we take the GEE approach in these 
analyses, since our focus in this pilot study is on the extent to which Let’s Know! affects 
proximal and distal child language outcomes, and not on the distribution of teacher-specific 
effects. 

Particularly for the negative binomial model, small sample sizes can be problematic for 
estimation of parameters and their standard errors, and thus for statistical tests and their 
interpretation (Aban et al. 2008; UCLA: Statistical Consulting Group 2014). In general, the 
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impact of small samples also affects the ability to reliably detect differences in means or 
average counts between three or more intervention groups, which requires larger sample 
sizes for well-powered comparisons relative to a two-group design. Thus, we approached 
our assessment of CBM differences between the B AU, Deep, and Broad versions using two 
strategies: (1) statistical significance testing based on a = 0.05 coupled with reporting of 
effect sizes and their interpretation; and (2) interpretation of substantive effect size cal¬ 
culations for effects under a more liberal p < 0.10 criteria (we note that all /^-values for 
interpreted effects in the results reported below were <0.08). For small-samples research 
including pilot studies, Kramer and Rosenthal (1999) emphasize the importance of effect 
sizes over attainment of a discrete non-significant/significant p value (such as a = 0.05); 
we feel this approach is informative here. Readers are cautioned about strict interpretation 
of p-values alone, and we note that the confirmation of the practical importance of our 
findings through replication is the goal of future follow-up randomized trials. Due to 
sample size limitations, treatment group was the only predictor in our models and no other 
covariates were included. We used dummy codes with BAU as the referent group, and we 
report and interpret contrasts between the Deep and Broad versions of the curriculum as 
well as effect size estimates for all regression parameters. In the results section below we 
provide additional specific analysis details for each outcome. SAS 9.4 was used for all 
analyses, and our GEE models were fit using SAS PROC GENMOD. 1 


Results 

Here we analyze the CBMs to investigate our first hypothesis: that children who were 
exposed to Let’s Know! would significantly outperform children receiving typical class¬ 
room practices on measures of these skills post-intervention. Next, we analyze results from 
the LCM to investigate the second hypothesis: that children exposed to Let’s Know! would 
significantly outperform children receiving typical classroom practices on a general 
measure of language comprehension post-intervention. We begin by presenting the 
descriptive statistics for all measures and testing for differences between the groups of 
children in the three conditions. 

Preliminary Analyses 

Table 3 contains descriptive statistics for the three proximal CBM assessments and the pre- 
and post-test LCM scores. Frequency distributions for these variables were visually ana¬ 
lyzed. Preliminary analyses tested initial equivalence among the three conditions on 
important teacher and child variables. Fisher’s Exact Tests indicated no differences among 
conditions in terms of teachers’ highest degree earned, years of preschool teaching 
experience, or certification to teach preschool (all p values >0.150). With respect to 
children, no differences were found in terms of age, F(2, 46) = 0.771, p = 0.468, gender, 
X 2 (2, N = 41) = 0.135, p = 0.935, minority status, x 2 (2, N = 40) = 4.433, p = 0.109, 
maternal education level, F(2, 35) = 0.867, p = 0.429, or LCM pre-test scores, F(2, 
46) = 0.745,/? = 0.481. 


1 Although the work is a collaborative effort, a member of the steering committee will take responsibility 
for the data and the accuracy of the data analysis. 
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Table 3 Descriptive statistics for proximal and distal measures 


Vocabulary 

Comprehension monitoring 

Text-structure knowledge 


n 

M SD 

Md n 

M 

SD 

Md 

n 

M 

SD 

Md 

Proximal measures 











Unit 1 (week 6) 











BAU 

16 

1.6 2.0 

1.0 16 

0.4 

0.8 

0.0 

16 

1.0 

0.7 

1.0 

Let’s Know! Deep 

18 

6.3 4.2 

5.5 18 

1.6 

1.6 

2.0 

18 

0.9 

0.7 

1.0 

Let’s Know! Broad 

15 

5.3 2.2 

6.0 15 

0.9 

1.3 

0.0 

15 

0.8 

0.8 

1.0 

Unit 3 (week 20) 











BAU 

15 

1.7 1.8 

1.0 16 

0.3 

0.6 

0.0 

16 

0.9 

0.7 

1.0 

Let’s Know! Deep 

17 

5.5 3.6 

5.0 17 

0.8 

1.3 

0.0 

17 

0.7 

0.6 

0.0 

Let’s Know! Broad 

12 

4.5 3.2 

6.0 12 

1.1 

0.9 

1.0 

12 

0.4 

0.5 

1.0 



LCM pre-test 



LCM post-test 





n 

M 


SD 

n 


M 


SD 

Distal measure 











BAU 


16 

4.3 


2.4 

15 


4.9 


2.3 

Let’s Know! Deep 


18 

3.8 


2.0 

17 


6.2 


2.1 

Let’s Know! Broad 


15 

3.3 


2.4 

12 


4.7 


3.1 


LCM listening comprehension measure 


Let’s Know! Impacts: Proximal Measures of Language Skills 

The first question addressed in this study concerned the extent to which Pre-K teachers’ 
implementation of Let’s Know!, which included the Broad and Deep versions, had positive 
impacts on children’s lower- and higher-level language skills. The primary outcomes of 
interest were children’s scores on the CBMs administered in week 6 (near the end of Unit 
1) and week 20 (near the end of Unit 3), which included items addressing vocabulary, 
comprehension monitoring, and text-structure knowledge. Table 3 contains the CBM 
scores for children in the three conditions. Note that for nearly all indices, the pattern of 
results showed highest scores for children whose teachers implemented the Deep version of 
Let’s Know!, followed by children whose teachers implemented the Broad version and then 
those whose teachers employed their normal classroom practices (BAU condition). 

Vocabulary 

Descriptive statistics and histograms indicated no extreme scores or outliers, but non¬ 
normality and positive skew were observed for both Unit 1 (M = 4.469, Var = 13.213, 
Skewness = 0.736) and Unit 3 (Af = 3.909, Var = 11.433, Skewness = 1.118). The 
shape of the vocabulary distribution was characteristic of discrete count data for which a 
Poisson regression or other count model is often applied. For the vocabulary data the 
variance was larger than the mean at both time points, suggesting overdispersion; potential 
sources of overdispersion in count outcomes include small within-group sample sizes, 
outliers, or clustering in the data (Hilbe 2011; Hox 2010; Raudenbush and Bryk 2002). 
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Counts with overdispersion are best modeled using a negative binomial distribution rather 
than Poisson (Cameron and Trivedi 2013; Coxe et al. 2009; Hilbe 2011; Long 1997). 
Parameter estimates for the negative binomial model can be exponentiated and interpreted 
as expected counts (for the intercept) and for each the predictor represents the change in 
expected counts per unit increase, all else held constant (Long 1997). Using dummy codes 
for the three instantiation groups, differences in expected vocabulary counts between BAU 
and the Deep version, and BAU and the Broad version, were directly estimated; we also 
used defined contrasts to detect differences between the Deep and Broad versions. 

The sphere of inference for this analysis was limited to the specific teachers included in 
the study, which was appropriate given that teachers volunteered for the pilot study, and 
some teachers had previously contributed to earlier work regarding the curriculum 
development; thus random selection and random assignment was not pure. In Table 4 we 
report regression coefficients, standard errors, and effect sizes (rate ratios) for the GEE 
analyses. Results show that both the Deep and Broad versions of Let's Know! had sig¬ 
nificantly greater expected counts on the vocabulary assessment compared to the BAU 
comparison, for both unit tests. For Unit 1, the estimated count of correct vocabulary words 
in the comparison group was exp(0.5089) = 1.663, which was not statistically different 
from zero (p = 0.1813). Note that this estimated count was similar to the mean count in the 
BAU condition (Table 4, M = 1.6 for BAU). Relative to children in the BAU condition, 
children of teachers in the Let’s Knowl Deep version correctly identified 3.8 times more 
vocabulary words (exp(1.3360) = 3.804); this represents an increase of about 280 % 
(100 % x (exp(5) — 1). That is, the mean count for the Deep version can be estimated as 
1.663 x 3.804 = 6.326, corresponding to the observed mean of 6.3 in Table 4. Children in 
the Broad version correctly identified 3.2 times more vocabulary words 
(exp(1.1525) = 3.166) than children in the BAU group (about a 217 % increase). There 
was no significant difference in the correct response rates for vocabulary words between 
the Deep and Broad versions, ^ 2 (1) = 0.93, p = 0.3342. 

A similar pattern of results was observed for Unit 3, for which the estimated count of 
correct vocabulary words in the BAU group was exp(0.5239) = 1.689, which was not 
statistically different from zero (p = 0.1689). Relative to children in the BAU condition, 
children of teachers in the Deep version correctly identified 3.2 times more vocabulary 
words (exp(1.1739) = 3.235), representing an increase of 224 %; and children in the 


Table 4 Negative binomial GEE regression predicting vocabulary by instantiation at Unit 1 and 3 


Variable 

B 

SE 

P 

Exp(B) a 

95 % Cl RR 

Lower 

Upper 

Unit 1 

Intercept 

0.5089 

0.3807 

0.1813 

1.663 

0.789 

3.508 

Let’s Know! Deep 

1.3360 

0.3983 

0.0008 

3.804 

1.742 

8.303 

Let’s Know! Broad 

1.1525 

0.4029 

0.0042 

3.166 

1.437 

6.974 

Unit 3 

Intercept 

0.5239 

0.3746 

0.1619 

1.689 

0.810 

3.510 

Let’s Know! Deep 

1.1739 

0.3836 

0.0022 

3.235 

1.525 

6.860 

Let’s Know! Broad 

0.9887 

0.4718 

0.0361 

2.688 

1.066 

6.775 


BAU is referent condition 
a Exp(B) = RR = rate ratio 
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Broad version correctly identified 2.688 times more vocabulary words 
(exp(0.9887) = 2.688; 169 % increase) than children in the BAU group. Similar to Unit 1, 
there was no statistically significant difference in the correct response rates for vocabulary 
words between the Deep and Broad versions, ^ 2 (1) = 0.41, p = 0.5196. 

Comprehension Monitoring 

Table 5 shows the distribution of responses for the comprehension monitoring (CM) task 
across both units and all three treatment groups. Given the ordinal nature of the data, we 
first fit a proportional odds regression model, predicting the cumulative probability of 
responses in higher CM categories relative to probability of scores in the lower CM 
response categories (O’Connell 2006). As with the vocabulary analyses, we used GEE 
estimation for robust standard errors to account for residual correlation within teachers, and 
dummy codes were used with BAU as the referent category. Results are provided in 
Table 6. We note that the coefficients a4 through otl are used to define intercepts for each 
cumulative representation of the ordinal outcome, and focus our interpretation on the 
treatment conditions. 

These analyses suggest a slight effect of responses being in the higher categories rel¬ 
ative to the BAU condition for children in the Deep version in Unit 1 {B = 1.6800, 
p = 0.0752) and similarly for children in the Broad version for Unit 3 (B = 1.8702, 
p = 0.0697). Whereas neither effect reached conventional levels of statistical significance 
(i.e., p < 0.05), due to the small sample size we interpret differences substantively using 
odds ratios for effects based on a more liberal criteria of ot < 0.08. Thus, for Unit 1, 
children in the Deep version were 5.366 times as likely (exp(1.68) = 5.366) to be in higher 
categories, relative to the BAU children; and for Unit 3, children in the Broad version were 
6.490 times as likely (exp(1.8703) = 6.490) to be in higher response categories, relative to 
the BAU group. These represent considerable improvements for children in these inter¬ 
ventions. Contrasts comparing cumulative responses for the Deep and Broad versions were 
not statistically significant for either unit (Unit 1 y?{l) = 0.17, p = 0.6776; Unit 3 
X 2 (l) = 0.03, p = 0.8639). 

After inspection of the response frequencies in Table 7, and coupled with the large 
effect sizes observed in the ordinal regressions for the Deep version at Unit 1 and the Broad 


Table 5 Comprehension monitoring and text structure frequency distributions across Units 1 and 3 for the 
three curricular groups 


Response 

score 

BAU Deep 

Unit 1 CM 

Broad 


Total 

BAU Deep 

Unit 3 CM 

Broad 

Total 

0 


12 


8 

9 


29 

13 

12 

4 

29 

1 


3 


0 

1 


4 

2 

0 

3 

5 

2 


0 


6 

4 


10 

1 

3 

5 

9 

3 


1 


0 

0 


1 

0 

1 

0 

1 

4 


0 


4 

1 


5 

0 

1 

0 

1 


Unit 1 TS 






Unit 3 TS 




0 

4 


5 

6 


15 


5 

6 

7 

18 

1 

8 


9 

6 


23 


8 

10 

5 

23 

2 

4 


4 

3 


11 


3 

1 

0 

4 
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Table 6 GEE ordinal regression results for comprehension monitoring, Units 1 and 3 


Variable 

B 

SE 

P 

Exp(B) a 

95 % Cl RR 

Lower 

Upper 

Unit 1 







a 4 

-3.1979 

0.6640 

<0.0001 




oe 3 

-2.9824 

0.7727 

0.0001 




a 2 

-1.6154 

0.5522 

0.0034 




OCi 

-1.2215 

0.5653 

0.0307 




Let’s Know! Deep 

1.6800 

0.9433 

0.0752 

5.366 

0.843 

34.15 

Let’s Know! Broad 

0.8041 

0.7056 

0.2544 

2.235 

0.561 

8.91 

Unit 3 







a 4 

-4.8901 

1.2252 

<0.0001 




oe 3 

-4.1784 

1.0346 

<0.0001 




a 2 

-2.1429 

0.6421 

0.0008 




Oil 

-1.5300 

0.7122 

0.0317 




Let’s Know! Deep 

0.8931 

0.9209 

0.3321 

2.443 

0.402 

14.850 

Let’s Know! Broad 

1.8702 

1.0310 

0.0697 

6.490 

0.860 

48.955 


version at Unit 3, we decided to conduct an additional test, comparing the probability of 
responses in the two lowest CM categories (0 and 1) to the probability of responses in the 
higher categories (2, 3, or 4). This approach has been advocated as an alternative when few 
responses are beyond 0 or 1, as we have here (Cameron and Trivedi 2013). Thus, we used 
GEE through logistic regression and a binary response of being in the three higher cate¬ 
gories or not; results are provided in Table 7. Although children in either Let’s Know! 
versions tended to score in higher CM response categories at Unit 1, we found a statisti¬ 
cally significant increase ( p < 0.05) only for those in the Deep version compared to BAU 
children (B = 2.93, p = 0.0226, OR = 18.75); this pattern held for those children in the 
Broad version only under a more liberal significance level (p < 0.08) (B = 2.015, 
p = 0.0709, OR = 7.50). For Unit 3, children in either Deep or Broad tended to have 
higher CM responses relative to the BAU condition based on p < 0.08, but neither attained 
statistical significance at conventional levels (for Deep, B = 1.83, p = 0.1199, 
OR = 6.23; for Broad, B = 2.37, p = 0.0747, OR = 10.70). 


Table 7 Logistic regression (GEE) predicting CM response categories 2, 3, or 4 


Variable 

B 

SE 

P 

Exp(B) a 

95 % Cl OR 

Lower 

Upper 

Unit 1 







Intercept 

-2.7081 

1.0022 

0.0069 




Let’s Know! Deep 

2.9312 

1.2858 

0.0226 

18.75 

1.508 

233.04 

Let’s Know! Broad 

2.0149 

1.1155 

0.0709 

7.50 

0.84 

66.77 

Unit 3 







Intercept 

-2.7081 

1.0022 

0.0069 




Let’s Know! Deep 

1.8326 

1.1783 

0.1199 

6.23 

0.621 

62.929 

Let’s Know! Broad 

2.3716 

1.3305 

0.0747 

10.70 

0.790 

145.34 
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Text-Structure Knowledge 

Similar to comprehension monitoring knowledge, responses to text-structure were ordinal 
(Table 5) and thus were modeled using a proportional odds regression analysis (O’Connell 
2006) with GEE estimation and robust standard errors to account for residual clustering, 
and dummy coding of the curriculum groups. The probability of being in cumulatively 
higher categories, relative to being in lower categories, was assessed. There were no 
significant differences across the versions for either Unit 1 or Unit 3 (all ps > 0.10). 
Similar to the other CBM analyses, contrast comparisons indicated no significant differ¬ 
ences between Deep and Broad for either Unit. 

Let's Know! Impacts: General Language Comprehension 

The second question addressed in this study concerned whether pre-K teachers’ imple¬ 
mentation of Let’s Know! improved children’s general language comprehension, based on 
the LCM. This was considered a distal outcome of the intervention, as general language 
comprehension was not directly targeted in the intervention; rather, Let’s Know! is 
designed to improve component language skills. This research question was addressed via 
analysis of covariance using children’s pre-test scores as a covariate in a regression pre¬ 
dicting post-test scores on the LCM. LCM pre-test scores were grand-mean centered and 
treated as a fixed-effect covariate. As in the previous analyses, condition was entered into 
the analysis as a series of dummy codes representing the two Let’s Know! conditions, and 
BAU served as the referent group. Also similar to the previous models, we used GEE and 
robust standard errors to account for residual correlation given children nested within 
teachers. For this continuous outcome, the proportion of variance between teachers for the 
post-test LCM scores was minimal; the intraclass correlation was less than 0.02. 

The descriptive statistics for pre- and post-test scores on the LCM were presented earlier 
in Table 3. GEE results are provided in Table 8. After adjusting for pre-test scores on the 
LCM, children whose teachers implemented the Deep version had significantly greater 
post-test LCM scores as compared to children in the BAU condition, with a large effect 
size (.B = 1.963,/? < 0.0001, d = 0.63). However, this was not the case for children whose 
teachers implemented the Broad version; the effect size was small and the comparison was 
not significant (.B = 0.382, p = 0.4424, d = 0.12). A contrast comparing the two Let’s 
Know! versions found no significant difference in adjusted post-test LCM scores between 
children whose teachers implemented the Broad versus Deep versions. 


Table 8 GEE analysis of covariance predicting post-test LCM from condition and pre-test LCM (grand- 
mean centered) 


Variable 

B 

SE 

P 

d b 

95 % Cl (. B ) 


Intercept 

4.3692 

0.1897 

<0.0001 


3.9974 

4.7410 

Let’s Know! Deep 

1.9630 

0.2795 

<0.0001 

0.63 

1.4151 

2.5108 

Let’s Know! Broad 

0.3819 

0.4971 

0.4424 

0.12 

-0.5924 

1.3561 

Pre-test LCM a 

0.6372 

0.0980 

<0.0001 


0.4450 

0.8294 


a Covariate is grand-mean centered 

b Uses SD (pre-test LCM control ) as conservative denominator for Cohen’s d 
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Discussion 

The Let’s Know! curriculum supplement was developed over two years using an iterative 
process of testing and revising based on careful observations of implementation and teachers’ 
feedback. Many language interventions focus primarily on lower-level language skills, such as 
vocabulary (Assel et al. 2007; Jackson et al. 2006; Justice et al. 2008; Lonigan et al. 2013; 
Mashbum et al. 2010; Piasta et al. 2010; Powell et al. 2010). Although these lower-level 
language skills have been linked to language and reading comprehension (Gough and Tunmer 
1986), higher-level language skills, such as comprehension monitoring and text-structure, also 
play a crucial role in literacy development (Oakhill and Cain 2012; Williams 2005; Williams 
et al. 2004, 2005). Let’s Know! aims to improve both lower- and higher-level language skills 
(vocabulary and comprehension monitoring/test-structure knowledge, respectively) as well as 
general language comprehension. This study explored the influence on pre-K children’s oral 
language skills of two versions of a language intervention ( Let’s Know! Broad, Let’s Know! 
Deep) relative to a BAU condition. Four main findings emerged: (1) both Let’s Know! versions 
improved vocabulary relative to the BAU condition, (2) both Let’s Know! versions appeared to 
have positive impacts on comprehension monitoring relative to the BAU condition, (3) neither 
Let’s Know! version improved text-stmcture knowledge relative to the BAU condition, and (4) 
only Let’s Know! Deep improved general language comprehension. 

First, we measured the proximal influence of the language intervention on vocabulary, a 
low-level language skill, and comprehension monitoring and text-structure, high-level 
language skills. For Units 1 and 3, children in both the Let’s Know! Deep and Broad 
conditions demonstrated more vocabulary knowledge than children in the BAU condition. 
Performance did not differ for children in the two Let’s Know! conditions. This study adds 
to the mixed findings regarding the effectiveness of language interventions at improving 
oral language, specifically vocabulary. Although some studies have not found improve¬ 
ments in vocabulary as a result of a language intervention or curriculum (Powell et al. 
2010), the results of this study align with the few studies demonstrating that language 
interventions or curriculums can improve vocabulary (Assel et al. 2007; Clarke et al. 
2010). Moreover, we demonstrated that vocabulary improvements may be possible on a 
proximal vocabulary measure after only 6 weeks of intervention whereas previous work 
primarily reports vocabulary results after completion of a longer intervention (e.g., 
20 weeks for Assel et al. 2007 and one school year for Clarke et al. 2010). 

The second main finding was that there was an improvement in children’s compre¬ 
hension monitoring after Unit 1 for the Deep version and after Unit 3 for the Broad version, 
relative to children in the BAU group. Importantly, neither version reached a 0.05 sig¬ 
nificance level, despite the large effect sizes. With the logistic regression results comparing 
the lower scores with the higher scores, however, we see that performance in Unit 1 for the 
Deep version did reach <0.05 significance and the performance in the Unit 3 Broad version 
remained at a 0.08 significance level. Thus, we found a pattern of improvement for chil¬ 
dren in both intervention conditions, but these results should be interpreted cautiously due 
to the small sample size. Finding this benefit from the intervention is particularly inter¬ 
esting considering the small sample size and that prior research has found successful 
comprehension monitoring of oral or written text begins to develop between preschool and 
third grade, with several studies reporting little success for younger children within this age 
range (see Wagoner 1983 for review). Thus, although we found that pre-K children’s 
scores improved with this language intervention, the scores may have improved more if the 
intervention was implemented with slightly older children. 
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Third, there was no difference in text-structure knowledge between children in the Let's 
Know! Deep, Let's Know! Broad, and BAU conditions. This was somewhat unexpected 
because of the intervention’s focus on text-structure, specifically in the Broad version. 
Similar to comprehension monitoring development, successful understanding of text- 
structure is generally found for older children or a subset of younger children who have 
good verbal skills and good working memory, which speaks to the difficulty level of this 
skill (Cain et al. 2004). The different results for vocabulary and text-structure may be 
related to the structure of the Words to Know lessons, which involved providing both 
explicit definitions of new words and implicit connections between words. However, Text 
Mapping and Integration lessons were not able to provide similarly explicit definitions and 
information. Perhaps the less explicit instruction did not allow for children to fully develop 
their understanding of text-structure (and the same may be true for comprehension mon¬ 
itoring), which could account for the lower performance in the related abilities measured in 
the CBMs. Thus, the introduction of the Let's Know! curriculum supplement, in either the 
Deep or Broad form, improved children’s vocabulary skills, a lower-level language skill, 
above that of attending a typical classroom. Implementation of Let's Know! improved 
comprehension monitoring abilities, a higher-level language skill, but neither version of 
the intervention improved text-structure knowledge. 

Fourth, we examined the intervention’s effect on language comprehension. Children 
whose teachers implemented the Let’s Know! Deep curriculum supplement showed greater 
gains on the Listening Comprehension Measure relative to children with teachers in the 
BAU condition. No difference in gains was found between participants in the Let’s Know! 
Broad condition compared to either Let's Know! Deep or BAU. Thus, our language 
intervention was successful in boosting children’s language comprehension abilities 
compared to standard classroom activities. Specifically, the Let's Know! Deep version, 
which allowed children increased practice opportunities on a restricted number of lesson 
types, went beyond immediate language improvement to influence general language 
comprehension. This pattern of higher scores in the Deep version was also found for the 
proximal measures of vocabulary and one unit of comprehension monitoring. Similar 
advantages of a “deep” teaching method have been reported for other areas of education, 
specifically science education (Schwartz et al. 2009). In a large study exploring the relation 
between deep or broad science training in high school and performance in college science 
courses, Schwartz and colleagues found a general advantage for “deep” training in high 
school leading to success in college science courses. However, it is difficult to compare the 
results of a curriculum for pre-K students and high school students. It is also important to 
note that even the Broad version seems promising in that it resulted in improved perfor¬ 
mance relative to the pre-test and for vocabulary and comprehension monitoring. Further 
research may find that the Broad version of the program is also effective at increasing 
language comprehension, but it currently seems that the Deep version shows the most 
promise upon which we may build an even more successful language intervention. An 
alternative explanation of these results is that the vocabulary boost found throughout the 
intervention led to improvements in general language abilities, perhaps due to having a 
larger vocabulary and a better understanding of how to acquire new words. Unfortunately, 
it is not within the scope of this paper to explore that relationship more deeply. 

Several limitations of this study warrant note. First, the teachers’ assignment to con¬ 
ditions was not wholly randomized, potentially leading to teachers having varying levels of 
experience with the curricula, as a result we must be cautious in the interpretation of the 
findings. Random assignment of teachers would increase our confidence that the results 
reflect treatment condition and not other variables. Second, the sample size was modest. 
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Future examination of effects of this intervention on children’s language skills is needed. 
Third, measures used to assess children’s outcomes were experimental. Future studies 
should examine treatment effects on standardized and/or commonly used measures. 
Finally, this study focused on children with low-SES backgrounds. Research with children 
from high-SES backgrounds may find different, potentially stronger results. 

To summarize, this study is an important contribution to the literature on pre-K chil¬ 
dren’s language development because it demonstrates that it may be possible to improve 
the vocabulary and comprehension monitoring abilities of low-SES pre-K children after 
just 6 weeks of curriculum intervention and general language comprehension abilities after 
just 21 weeks. Moreover, this work suggests that with this language-focused curriculum, 
implementing fewer lesson types with more time for practice in each one (the Deep 
version) allows children to build general language comprehension skills better than if they 
had experienced a larger variety of lessons (the Broad version). These data from an “as 
treated” design will inform the development of a follow-up randomized trial study further 
testing the impact of the Let's Know! intervention. Taken together, the results of this study 
show both proximal and distal advantages of the Let's Know! curriculum supplement for 
pre-K children and can inform future language research and curriculum development. 
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Appendix 1 


Let's Know! 

Animals 

Words To Know 

PreK 

Compare and Contrast 

Lesson 3 


Show Me What you Know! Animals live in many different kinds of homes! We’re going to be animal 
biologists reporting on how animal homes are alike and different! 


Teaching Objective(s): 

• Use the Words to Know in hands-on activities providing an example of a word's meaning 


Teaching Technique(s): 

• Rich Instruction 

Lesson Materials You Provide: 

• Stuffed animals, animal figures, or 
puppets to act out ‘protect’ 

• Materials for tabbed books - scissors, 
ruler, stapler, paper 

o 1 tabbed book per student 

Unit Materials Provided: 

• Habitat pictures from Lesson 1 

• Pictures to demonstrate alike and 
different 

• Pictures to glue on a page for their tabbed 
word book - habitat, alike, different, 
protect 

• Vocabulary picture cards (optional) 

Lesson Text: 

• Hames-af-hi yin g Thin gs by Bobbie 
Kalman 

Talk Structure for We Do/You Do: 

• Think-Pair-Share 


Words to Know: 

• habitat (noun): the area where an animal likes to 
live - H, A 

• alike (adjective): two things that are like each 
other - 0 

• different (adjective): not the same or unlike - A 

• protect (verb): to keep safe - H, A 

• shelter (noun): something that provides cover or 
protection - H, A 

• survive (verb): to stay alive - A 

• prairie (noun): large area of flat grasslands - H, A 

• insect (noun): bug with a head, thorax and 
abdomen and three pairs of legs - H 

Lesson Texts: 

• H = Homes of Living Things by Bobbie Kalman 

• A = Animal Ha b itats by Michelle Kramer 

• L = L Qud . 3n dJ)yjeLU- An _A q iin. al j:_0p.pi) J §itg^. B i pok 
by Lisa Bullard 

• 0 = other texts 


Special instructions for this lesson: 

• The 1 Do/We Do parts are combined in this lesson to allow focusing on one word at a time for 
modeling by the teacher and guided practice with students 

• In the text, Homes of Living Things, the four unit vocabulary words appear on the following pages: 
habitat-6,7,8,15,16, & 17 protect-4 & 11 different-5 

• In each of the four Words to Know lessons in the unit, the students will create pages for a tabbed book 
on Animal Homes Words to Know. In this lesson students will create a page illustrating one of the first 
four words. This can be done later at a center time or free choice. For help with creating a tabbed 
book, visit http://www.squidoo.CQm/tabbed-book 
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Lesson Plan 


Set 

Introduce the idea that there are lots of words that people use, and sometimes we don’t 
know what these words mean. When we are reading a non-fiction or information book, 
there might be many new words about the book topic. We have to train our brains to 
listen for words we don’t know, and then try to figure out what they mean. 

You could say: "There are so many words that people use every day and we don't always know 
what thev mean. In our new book Homes of Living Things, there are new words about animals 
and where they live. We have to be super animal biologists and train our brains to listen for these 
new words so we will be able to learn about animal homes and report to our friends lots of new 
and interesting information." 

IDO/ 

We Do 

Follow the steps for Rich Instruction for four of the unit’s Words to Know: habitat, protect, 
alike, and different. 

• Introduce the words while reading the text Homes of Living Things, studying the 
WRAP vocabulary pictures, reading another text, or using any other method 

• Follow the steps for rich instruction for each word 

For the word 'habitat' you could say: (p. 15) "Our book says, 'Chimpanzees live in hot, rainy, 
forest habitats.' A habitat is the area where an animal likes to live. At the zoo they build special 
habitats for the different animals. I like to visit the African grasslands habitat. It has giraffes, 
water buffaloes, and large birds. Let's all say the word habitat together." 

Display habitat pictures from Lesson 1 and ask students to name the type of habitat 
depicted. 

For the word ‘protect’ you could say: (p. 11) "This says the rattlesnake's home is a cave. 'The 
cave protects the snake from the hot sun.' Protect means to keep safe. The sun in the desert can 
be very, very hot. Many animals, including people, need to be protected from the sun. My mom 
and dad cross the street with me so I will be protected from the cars. Let's all say the word 
protect" 

Give students a turn to act out protect with a play animal and props. 

For the word 'alike' you could say: (pp. 12 & 13) "The homes of these animals are alike 
because they are all homes made from holes in something. 'Alike' means things that are like each 
other. My friend and I have bikes that look the same; they're alike. My grandpas both have white 
hair and look alike. Let’s all say the word alike." 

Show students 'alike' cards, and students say 'alike.' 

For the word 'different' you could say: (p. 24) "These pictures show different animal homes. 
Different means not the same or unlike. My hair is a different color than my daughter’s. My 
friend and I ride different buses to school. Let’s all say the word different." 

Students turn over two animal cards and say 'different.' 


Springer 










Child Youth Care Forum (2016) 45:367-392 


389 


You Do 

Students illustrate or glue a picture on a page to depict protect, habitat, alike or different 
for their tabbed Words to Know book. 

You could say: "We are going to learn lots of new words while we study animal homes. Here is a 
Words to Know book for each of you. Today I want you to make a page with one of our new 
words. We can add to our books as we learn new words and use them to help us write our 
reports about how animal homes are alike and different." 

Ask students to share their page with a partner or the rest of the class. 

Close 

Summarize the importance of learning to identify new words that occur around us and to 
learn more about these words. Explain how important it is to be able to identify words in 
books we read so we can learn more information about the world around us. 

You could say: "Nice job learning four new words today. We are going to hear these words all 
through our unit on Animal Homes. It is really important to pay attention to words and what they 
mean. That helps us understand new information about animals and Animal Homes. Together 
let's practice using each of our new words in a sentence: 

• Some people protect elephants. 

• Drink water in a desert habitat. 

• The two birds’ nests were alike. 

• The horses are two different colors." 


Appendix 2 

See Table 9. 


Table 9 Lesson sequence within a Let’s Know! unit: Broad and Deep versions 


Week 

Lesson 

Broad version 

Deep version 

1 

1 

Hook 

Hook 


2 

Read to me 

Read to me 


3 

Words to know 

Words to know 


4 

CBM preview 3 

CBM preview 

2 

5 

Text mapping 

Words to know 


6 

Words to know 

Words to know practice 


7 

Integration 

Integration 


8 

Read to know 

Integration practice 

3 

9 

Read to me 

Read to me 


10 

Text mapping 

Integration 


11 

Integration 

Words to know practice 


12 

Words to know 

Words to know 
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Table 9 continued 

Week 

Lesson 

Broad version 

Deep version 

4 

13 

Text mapping 

Integration 


14 

Integration 

Integration practice 


15 

Words to know 

Words to know 


16 

Read to know 

Words to know practice 

5 

17 

Read to me 

Read to me 


18 

Text mapping 

Integration 


19 

Integration 

Integration practice 


20 

Read to know 

Words to know practice 

6 

21 

Read to know 

Integration practice 


- 

CBMs 

CBMs 

7 

22 

Stretch and review 

Stretch and review 


23 

Stretch and review 

Stretch and review 


24 

Close 

Close 


a CBM curriculum-based measure developed for the Let’s Know! curriculum supplement. Teachers sample 
children’s skill on the CBM early in a unit in a group-administered lesson (lesson 4, week 1); the CBMs 
implemented in week 6 are individually administered and are followed by two “stretch and review” lessons 
in which teachers can review select content 
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